Detailed Problem Descriptions for General Program Synthesis Benchmark Suite
نویسندگان
چکیده
Recent interest in the development and use of non-trivial benchmark problems for genetic programming research has highlighted the scarcity of general program synthesis (also called“traditional programming”) benchmark problems. We present a suite of 29 general program synthesis benchmark problems systematically selected from sources of introductory computer science programming problems. This suite is suitable for experiments with any program synthesis system driven by input/output examples. We present results from illustrative experiments using our reference implementation of the problems in the PushGP genetic programming system. This technical report provides sufficient detail of the problems and our reference implementation for researchers to implement and attempt to solve these problems in other synthesis systems. The results show that the problems in the suite vary in difficulty and can be useful for assessing the capabilities of a program synthesis system.
منابع مشابه
BenchmarX
Bidirectional transformation (BX) is a very active area of research interest. There is not only a growing body of theory, but also a rich set of tools supporting BX. The problem now arises that there is no commonly agreed-upon suite of tests or benchmarks that shows either the conformance of tools to theory, or the performance of tools in particular BX scenarios. This paper sets out to improve ...
متن کاملSynthesis of Domain Specific CNF Encoders for Bit-Vector Solvers
SMT solvers are at the heart of a number of software engineering tools. These SMT solvers use a SAT solver as the back-end and convert the high-level constraints given by the user down to low-level boolean formulas that can be efficiently mapped to CNF clauses and fed into a SAT solver. Current SMT solvers are designed to be general purpose solvers that are suited to a wide range of problems. H...
متن کاملA Benchmark Suite for Hybrid Systems Reachability Analysis
Since about two decades, formal methods for continuous and hybrid systems enjoy increasing interest in the research community. A wide range of analysis techniques were developed and implemented in powerful tools. However, the lack of appropriate benchmarks make the testing, evaluation and comparison of those tools difficult. To support these processes and to ease exchange and repeatability, we ...
متن کاملEf®cient solution strategies for building energy system simulation
The ef®ciencies of methods employed in solution of building simulation models are considered and compared by means of benchmark testing. Direct comparisons between the Simulation Problem Analysis and Research Kernel (SPARK) and the HVACSIM programs are presented, as are results for SPARK versus conventional and sparse matrix methods. An indirect comparison between SPARK and the IDA program is ...
متن کاملEfficient Solution Strategies for Building Energy System Simulation
The efficiencies of methods employed in solution of building simulation models are considered and compared by means of benchmark testing. Direct comparisons between the Simulation Problem Analysis and Research Kernel (SPARK) and the HVACSIM+ programs are presented, as are results for SPARK versus conventional and sparse matrix methods. An indirect comparison between SPARK and the IDA program is...
متن کامل